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An Early Look at the Effects of 
Success Academy Charter Schools 


Success Academy is a rapidly expanding charter school network in New York City, with 
schools located in the Bronx, Brooklyn, Manhattan, and Queens. In the 2016-2017 school year, 
Success Academy served roughly 14,000 students across 41 elementary, middle, and high 
schools, which at the time was about 13 percent of the students attending charter schools in the 
city and equivalent to about 1.2 percent of all New York City Department of Education 
(NYCDOE) students. 1 

In this paper, MDRC looks back to the beginning of Success Academy and uses natu¬ 
rally occurring lotteries for the seven Success Academy schools in operation during the 2010- 
2011 school year to estimate the difference in students’ academic achievement caused by the 
opportunity to enroll in a Success Academy school. This work is part of an independent analysis 
that MDRC conducted for Success Academy’s 2015 Investing in Innovation (i3) federal grant 
application. The educational outcomes of elementary students who received placements in their 
local Success Academy schools by being selected in lotteries are compared with the outcomes 
of students who entered these lotteries but were not selected for their local Success Academy 
schools. Because not all students selected in lotteries enrolled in Success Academy schools and 
some of those not selected through lotteries for their local (or closest) Success Academy school 
ended up enrolling in a different Success Academy school, MDRC also estimates the effects of 
enrolling in a Success Academy school (for the lottery sample). A statistical adjustment makes 
it possible to compare those who won their lotteries and did enroll in Success Academy schools 
with those who lost their lotteries and did not enroll in Success Academy schools. 

The preliminary analysis in this paper focuses on the 4,710 students (3,804 students en¬ 
tering kindergarten and 906 entering first grade) who applied to oversubscribed Success 
Academy schools in the spring of 2010 for the 2010-2011 school year and who therefore 
participated in lotteries. Only applicants from this round of admissions could be followed long 
enough to measure their academic achievement using New York State’s third- and fourth-grade 
mathematics and reading exams at the time the analysis was conducted. While 4,710 students is 
a relatively large sample, because the number of applicants greatly exceeded the number of 


'Data on Success Academy schools are taken from the organization’s website. See Success Academy 
(2017). According to the New York City Charter School Center (2017), a total of 106,600 students were 
enrolled in New York City charter schools in the fall of 2016. According to the New York City Department of 
Education (2017), a total of 1,133,963 students were enrolled in public schools citywide in 2015-2016, 
excluding charter schools. For current information about Success Academy and its educational offerings see 
www.successacademies.org. 
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seats available for the 2010-2011 school year, the sample only includes about 700 students who 
were offered admission. The resulting low ratio of lottery “winners” to all applicants limits the 
statistical precision of the estimates presented. 2 

Within the constraints of this analysis, MDRC finds that for the subset of students who 
were selected through a lottery and then enrolled in a Success Academy school, enrollment had 
large, positive effects on students’ math achievement in third grade, roughly equivalent in 
magnitude to between one and one and a half typical years of learning for this age group. 
Effects on reading did not reach conventional levels of statistical significance but are positive 
and promising. A larger sample size would be necessary to obtain more precise estimates. As 
explained below, the program’s effect was also estimated for all lottery winners, not just those 
who enrolled, and these estimates were similar but of a smaller magnitude. 

The remainder of this paper explains these findings in more detail. Appendix A de¬ 
scribes how the lotteries occur within the Success Academy admission process and how the 
analytic models are estimated. 


Estimating the Effects of Success Academy 

When considering the effects of social policies and educational programs, researchers 
often estimate both the effect of receiving an offer to participate in a particular program and the 
effect of actually receiving the program. Policymakers may be interested in the effects of 
offering a program, since that is within their control, while others may be more interested in the 
effects of the program on the people who chose to respond to the offer. Here MDRC estimates 
both (1) the effect of winning the opportunity to attend a Success Academy school and (2) the 
effect of winning the offer to attend a Success Academy school and enrolling the following fall. 
Both of these estimates are important, and, in fact, the effect of enrolling is estimated by 
performing a statistical adjustment on the effect estimated using the original random lottery 
sample. Box 1 describes the two different approaches in greater detail. 

Estimating the Effect of the Opportunity to Enroll in a Success Academy 

School 

To estimate the effect of the offer of a seat at a Success Academy school, MDRC used 
naturally occurring lotteries for oversubscribed Success Academy schools to identify two 


2 Because of the preliminary nature of this analysis and the limited power arising from the unusual ratio of 
lottery winners to nonwinners in the sample, this paper treats findings with p-values of 0.10 or less as statisti¬ 
cally meaningful when they are part of a pattern of positive effects. 
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Box 1 

Methods Used in This Analysis to Estimate Effects 

The method used in this analysis is based on a lottery, which, like a randomized controlled trial 
(widely considered to be the gold standard for studying the effects of educational, social, or 
medical innovations), randomly determines who is assigned to a program and who is not and 
creates two groups that are the same on average in all ways, except for their assignment to the 
program. Any future differences in the average outcomes of the two groups thus can be 
attributed to the program or innovation. Comparing the outcomes of lottery winners and lottery 
losers is called an “intent-to-treat” analysis. Such an analysis meets the U.S Department of 
Education’s highest standard of evidence. Because the winner of a lottery could decide not to 
receive the intervention (in this case, attending a Success Academy school), or a lottery loser 
could receive the intervention by other means (in this case, by attending a different Success 
Academy school or later being admitted from the waiting list), it is also useful to estimate the 
average effect of winning a lottery and receiving the intervention, which in this study means 
enrolling in a Success Academy school, compared with not winning a lottery and not receiving 
the intervention. This latter estimate is called the compiler average causal effect (CACE). 


groups of comparable students: those who randomly won the opportunity to attend a Success 
Academy school (which in evaluation research would typically be referred to as a study’s 
“program group”) and those who randomly lost the opportunity to attend a Success Academy 
school (which would typically be referred to as the study’s control group). Because students 
were randomly assigned to these two groups, any difference in average outcomes between the 
two groups following the lottery can be attributed to the offer of attending a Success Academy 
school. (See Appendix A for a more detailed description of this analytic approach.) 

Estimating the Effect of Enrolling in a Success Academy School 

Students’ “local lottery” assignment results (that is, as explained in Appendix A, the re¬ 
sults for their nearest Success Academy schools) were not the final factor determining whether 
they actually attended Success Academy schools. Fifty percent of the Success Academy 
program group students decided not to attend Success Academy schools. There are various 
reasons why a student who won a Success Academy lottery may not have enrolled. For exam¬ 
ple, the Success Academy school may have been located too far from his or her home. (Some¬ 
times a lottery took place before a school’s final location was detennined.) A student may have 
applied to multiple charter schools and received an offer from a more preferred school. A 
student may have decided to attend a traditional elementary school. Or a student and his or her 
family may have decided not to enroll for another reason. Similarly, roughly 25 percent of the 
control group students enrolled in Success Academy schools, either by being assigned as part of 
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the typical assignment process (to schools farther away, after losing lotteries for their nearest 
schools) or by coming off local schools’ waiting lists to fill available spaces. Thus it is also 
useful to estimate the average effect of enrolling in a Success Academy school. This enrollment 
effect — the compiler average causal effect (CAGE) mentioned in Box 1 — is estimated using 
a standard application of two-stage least squares (2sLs) instrumental variables analysis, 3 an 
approach often applied in randomized experiments and lottery-based studies. 4 (See Appendix A 
for a more detailed description of the CACE approach.) 

It is important to note that while the intent-to-treat approach meets the U.S. Department 
of Education’s highest standard of evidence, the CACE approach provides weaker evidence, in 
part because it rests on untestable assumptions. 5 In addition, a note of caution should be heeded 
when interpreting the estimated effects of enrolling in a Success Academy school reported in 
this study. The effects of enrollment only apply to students who won lotteries and chose to 
enroll in Success Academy schools; the CACE analysis does not address whether or how 
Success Academy would have affected students who were offered the opportunity to attend but 
went elsewhere. 


Success Academy Applicants and Enrollees 

On average, Success Academy students are predominantly Black and Hispanic; the majority of 
the students qualified for free or reduced-price lunches. The sections below describe the 
characteristics of four samples of students relevant to the estimation of Success Academy’s 
effects: the students who applied to Success Academy schools, the students in the lottery sample 
used in this analysis, the students in the lottery sample who won seats in Success Academy 
schools and enrolled, and the students in the lottery sample for whom follow-up test scores are 
available. 

Applicants 

Students who applied to Success Academy schools in 2010 were largely economically 
disadvantaged. Specifically, roughly 70 percent of the Success Academy first-grade applicants 
in the sample qualified for free or reduced-price lunches. In comparison, 87 percent of all 
NYCDOE students in the neighborhoods surrounding Success Academy schools receive free or 


3 Angrist, Imbens, and Rubin (1996); Gennetian, Morris, Bos, and Bloom (2005). 

4 Abdulkadiroglu et al. (2011); Bloom and Unterman (2014); Gennetian, Morris, Bos, and Bloom (2005); 
Ludwig and Kling (2007). 

5 What Works Clearinghouse (2014a). 
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reduced-price lunches. 6 Like the students in the neighborhoods surrounding Success Academy 
schools, the majority of the students are Hispanic (53 percent) or Black (42 percent). 7 

Lottery Participants 

MDRC capitalized on the Success Academy admissions process to identify a spring 
2010 lottery sample of 4,710 students from 14 lotteries held for seven Success Academy 
elementary schools (3,804 students entering kindergarten and 906 entering first grade). When 
MDRC requested school records from NYCDOE those records were only available through the 
2013-2014 school year, so the present analysis could follow those 2010 applicants for four years 
after they participated in Success Academy lotteries (through third grade for students who 
participated in lotteries to enter kindergarten and through fourth grade for students who partici¬ 
pated in lotteries to enter first grade). The present analysis could not follow students who 
applied in later years. Table 1 provides more detail about the follow-up period for these two 
groups of applicants. 

Across all measurable background characteristics, on average, the Success Academy 
program group members (referred to in these tables as “lottery winners”) and their control group 
counterparts appear equivalent. 8 When there are small differences between the groups, these 
differences are not statistically significant at the 0.05 level, and do not warrant further sensitivity 
analyses. 9 Tables 2 and 3 indicate that, like the overall applicant sample, students in the sample 
are primarily Black and Hispanic and over half qualified for free or reduced-price lunches. 10 
About half of the sample is male and half is female. At least 20 percent of the students in the 
sample of Success Academy lottery participants also applied to at least one other charter school 
(not shown). 11 


6 This comparison relies on data from the Success Academy first-grade-entrant sample. A little over half of 
the Success Academy kindergarten applicants were reported as qualifying for free or reduced-price lunches. 
However, because Success Academy provides free lunch to all of its students, the families of Success Acade¬ 
my kindergarteners may be less motivated to turn in the required free lunch forms than the families of other 
students across the district. In contrast, NYCDOE collected the baseline data on free or reduced-price lunch for 
the sample of first-grade applicants while those students were attending non-Success Academy kindergartens. 

’“Students in the neighborhoods surrounding Success Academy schools’’ are defined as the students living 
in the areas identified by Success Academy as the schools’ “community school districts.” 

8 These results come from an omnibus test that regressed a binary indicator for program group status on all 
of the characteristics in the table. 

9 What Works Clearinghouse (2014a). 

10 As mentioned in footnote 6, the families of Success Academy kindergartners may be less motivated to 
turn in the required free lunch forms than the families of other students in the district. 

"Roughly equal percentages of students in the kindergarten and first-grade control groups enrolled in 
charter schools run by another network (about 20 percent in each case). 
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Table 1 


Evaluation Follow-Up Period 



Baseline 

Follow-Up 

Follow-Up 

Follow-Up 

Follow-Up 

Applicant group 

Year 

Year 1 

Year 2 

Year 3 

Year 4 

Kindergarten entrants 

NA 

K 

Grade 1 

Grade 2 

Grade 3 

First-grade entrants 

K 

Grade 1 

Grade 2 

Grade 3 

Grade 4 


NOTES: Students competed in lotteries for Success Academy schools in the spring of 
2010. Follow-Up Year 1 is the 2010-2011 school year. This study tracks students 
through the 2013-2014 school year. 

NA = not applicable. 


Table 2 


Baseline Characteristics of Kindergarten Entrants 




Control 


P-Value for 


Lottery 

Group 

Estimated 

Estimated 

Characteristic 

Winners Counterparts 

Difference 

Difference 

Race/ethnicity (%) 

Hispanic 

53.0 

53.9 

-0.9 

0.761 

Black 

41.8 

38.9 

2.8 

0.345 

Multiracial 

1.1 

2.4 

-1.3 

0.241 

White 

1.9 

2.2 

-0.3 

0.707 

Asian 

2.2 

1.9 

0.3 

0.729 

Male (%) 

57.3 

52.5 

4.8 

0.117 

Eligible for free/reduced-price lunch (%) 

50.7 

44.9 

5.8 

0.114 

Age 

4.2 

4.2 

0.0 

0.460 

Sample size 

400 

3,001 




SOURCES: MDRC's calculations use Success Academy school assignment data and New York City 
Department of Education school records and state test data for Success Academy applicants in the spring 
of 2010. 


NOTES: Values for Success Academy lottery winners are the simple means for all lottery winners. 

Values for the difference between Success Academy lottery winners and control group members are 
obtained from a regression of a given baseline characteristic on a series of indicator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. The 
coefficient on the latter indicator variable equals the difference in the mean baseline characteristic 
for lottery winners and control group members. The value for control group members equals the 
corresponding value for Success Academy lottery winners minus the estimated difference between lottery 
winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated 
as: ** = 1 percent; * = 5 percent. 
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Table 3 

Baseline Characteristics of First-Grade Entrants 




Control 


P-Value for 


Lottery 

Group 

Estimated 

Estimated 

Characteristic 

Winners Counterparts 

Difference 

Difference 

Race/ethnicity (%) 

Hispanic 

51.1 

53.7 

-2.7 

0.513 

Black 

42.8 

39.2 

3.6 

0.368 

Multiracial 

1.8 

0.6 

1.2 

0.184 

White 

2.5 

3.5 

-1.0 

0.511 

Asian 

1.8 

1.3 

0.5 

0.666 

Male (%) 

51.1 

55.8 

-4.8 

0.058 

Eligible for free/reduced-price lunch 

97.2 

96.3 

0.8 

0.528 

Age 

5.3 

5.3 

0.0 

0.568 

Sample size 

314 

592 




SOURCES: MDRC's calculations use Success Academy school assignment data and New 
York City Department of Education school records and state test data for Success Academy 
applicants in the spring of 2010. 


NOTES: Values for Success Academy lottery winners are the simple means for all lottery 
winners. Values for the difference between Success Academy lottery winners and control 
group members are obtained from a regression of a given baseline characteristic on a series of 
indicator variables that identify each lottery plus an indicator variable that equals 1 for lottery 
winners and 0 for lottery losers. The coefficient on the latter indicator variable equals the 
difference in the mean baseline characteristic for lottery winners and control group members. 
The value for control group members equals the corresponding value for Success Academy 
lottery winners minus the estimated difference between lottery winners and control group 
members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: ** = 1 percent; * = 5 percent. 


Enrollees 

To examine enrollment rates for students who won Success Academy lotteries, MDRC 
looked at participation in various Success Academy preenrollment activities that students (and 
their parents/guardians) who were offered a seat were told to attend. Specifically, after winning 
the lottery and before the school year started, students and their families were informed of a 
welcome meeting, student registration, a uniform fitting, and a dress rehearsal. In addition, 
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before enrollment (as is common among many charter networks), parents/guardians signed a 
contract highlighting the education-related activities offered throughout the year and outlining 
the various ways the Success Academy staff communicates with families. 

Of the lottery winners in the sample (both kindergarten and first-grade entrants), about 
82 percent attended a welcome meeting. Approximately 61 percent of lottery winners attended 
student registration, 54 percent attended a uniform fitting, and 50 percent attended a dress 
rehearsal. With few exceptions, lottery winners who did not attend an activity did not attend 
subsequent activities. Ultimately, about 50 percent of lottery winners enrolled in Success 
Academy schools in the 2010-2011 school year. 12 (As explained above, there are various 
reasons why a student might not have enrolled.) 

Students for Whom Test Scores Were Available 

While the Success Academy program and control group appear similar at the time of 
the lottery, this analysis focuses on the students within each group for whom test scores are 
available in third grade. As shown in Table 4, among both the kindergarten- and first-grade- 
entrant samples, math and reading test scores were available for at least 75 percent of both the 
Success Academy program group and control group members, and any differences in the rates 
of data availability were not statistically significant. 13 Based on standards set by the What 
Works Clearinghouse, the overall and differential levels of attrition (missing data) reported here 
are not substantial and do not warrant further sensitivity analyses. 14 In addition, the baseline 
equivalence seen after random assignment has been sustained for each follow-up analysis 
sample. 


In each follow-up year, students’ academic achievement was measured by their perfor¬ 
mance on the New York State reading and math exams. To facilitate comparisons over time, 
each student’s exam score was standardized as a z-score using each year’s New York State 
mean and standard deviation of test scores by grade and subject. 15 This standardization results in 


12 About 97 percent of those who enrolled in Success Academy schools attended all four of these preen¬ 
rollment activities. Compared with Success Academy lottery winners as a whole, on average fewer Hispanic 
students (by 6 percentage points) and more Black students (by 3 percentage points) attended Success Academy 
schools. 

13 When a sample member is missing test score data, it is probably because the student was held back, 
moved out of the New York City school district, or enrolled in a private or parochial school. 

14 What Works Clearinghouse (2014b). 

15 A z-score is a measure of the distance between a student’s test score and the New York State average, in 
standard deviations. For example, if a student’s exam score were exactly the same as the New York State mean 
score for that grade and subject, that student’s z-score value would be zero; if a student scored exactly one 

(continued) 



Table 4 

Test Score Availability 


Data Available (%) 

Control 

Lottery Group Estimated 

Winners Counterparts Difference 

P- Value for 
Estimated 
Difference 

Kindergarten entrants 





Third-grade mathematics 

75.61 

73.29 

2.32 

0.231 

Third-grade reading 

75.61 

71.18 

4.43 

0.301 

First-grade entrants 





Third-grade mathematics 

77.80 

74.70 

3.10 

0.241 

Third-grade reading 

78.80 

76.20 

2.60 

0.311 

Fourth-grade mathematics 

78.34 

78.86 

-0.52 

0.856 

Fourth-grade reading 

78.34 

78.86 

-0.52 

0.856 


SOURCES: MDRC's calculations use Success Academy school assignment data and New 
York City Department of Education school records and state test data for Success Academy 
applicants in the spring of 2010. 


NOTES: Values for Success Academy lottery winners are the simple means for all lottery 
winners. Values for the difference between Success Academy lottery winners and control 
group members are obtained from a regression of a given outcome on a series of indicator 
variables that identify each lottery, plus an indicator variable that equals 1 for lottery winners 
and 0 for lottery losers, plus a set of covariates: gender, age, free/reduced-price lunch status, 
race, and number of days absent in the baseline school year. The coefficient on the latter 
indicator variable equals the difference in the mean follow-up outcome for lottery winners 
and control group members. The value for control group members equals the corresponding 
value for Success Academy lottery winners minus the estimated difference between lottery 
winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels 
are indicated as: ** = 1 percent; * = 5 percent. 


estimated effects being reported in standard deviations, a measure often used to compare 
findings across studies that can be translated into months of regular school-year instruction 
based on the benchmarks in Hill, Bloom, Black, and Lipsey (2008). 


standard deviation higher than the New York State mean for the grade and subject, the student’s z-score would 
be one. 
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The Effects of Success Academy 

As mentioned above, this section first describes the effect of being offered the opportunity to 
enroll in a Success Academy school (the intent-to-treat effect), then describes the effect of 
actually enrolling in a Success Academy school (the compiler average causal effect). 

The Effect of Being Offered the Opportunity to Enroll 

Findings for the kindergarten-entrant sample in the first row of Table 5 indicate that, on 
average, in the fourth year of follow-up (that is, for most students, in third grade — the first year 
that students take state exams), being offered the opportunity to attend a Success Academy 
school increased students’ academic achievement in math; the effect on students’ academic 
achievement in reading is promising, though less clear. Specifically, winning a lottery produces 
an effect in math that is roughly equivalent in magnitude to an additional four months of 
learning for this age group (effect size = 0.179, p-value = 0.086). 16 Said differently, at the end of 
the fourth follow-up year, Success Academy program group members were roughly a third of a 
year ahead of students in the control group. Success Academy program group members in the 
kindergarten-entrant sample also appear to make substantial gains in reading (effect size = 
0.109, p-value = 0.185), but these estimates are not close to statistically significant, as reflected 
in the nonsignificant p-value, and it is not possible to rule out the possibility that the program 
did not have an effect. More precise estimates would require larger sample sizes. 

For the first-grade-entrant sample (Table 6), winning a lottery produces consistent third- 
grade and fourth-grade effects in math that are roughly equivalent in magnitude to an additional 
one-third to one-half of a year of learning for this age group (effect size = 0.185, p-value = 
0.040; and effect size = 0.267, p-value= 0.003, respectively). 17 Said differently, at the end of the 
third follow-up year, Success Academy program group members’ math scores were roughly a 
third of a year ahead of control group students’ scores; at the end of the fourth follow-up year 
they were roughly half a year ahead. The estimate of Success Academy’s effect on reading 
scores for the first-grade-entrant sample is positive but not statistically significant, and thus is 
less reliable (effect size = 0.148, p-value = 0.097). The estimate is especially unreliable in the 
fourth grade where the result is far from statistically significant (effect size = 0.078, p-value = 
0.362). 


16 According to Hill, Bloom, Black, and Lipsey (2008), the typical annual gain for third-graders in math is 
0.52 standard deviations; in reading it is 0.36 standard deviations. 

17 According to Hill, Bloom, Black, and Lipsey (2008), the typical annual gain for fourth-graders in math is 
0.56 standard deviations; in reading it is 0.40 standard deviations. 
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Table 5 


The Effects of Winning a Success Academy Lottery 
on Kindergarten Entrants After Four Years 



Estimated 

P- Value for 


Difference 

Estimated 

Outcome (%) 

in Standard Deviations 

Difference 

Third-erade test 



Mathematics 

0.179 

0.086 

Reading 

0.109 

0.185 


SOURCES: MDRC's calculations use Success Academy school assignment data and 
New York City Department of Education school records and state test data for Success 
Academy applicants in the spring of 2010. 

NOTES: This sample contains 400 lottery winners and 3,001 control group members. 

Values for the difference between lottery winners and control group members are 
obtained from a regression of a given outcome on a series of indicator variables that 
identify each lottery, plus an indicator variable that equals 1 for lottery winners and 0 
for lottery losers, plus a set of covariates: gender, age, free/reduced-price lunch status, 
race, and number of days absent in the baseline school year. The coefficient on the 
lottery winners indicator equals the difference in the mean follow-up outcome 
for lottery winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance 
levels are indicated as: ** = 1 percent; * = 5 percent. 


The Effect of Enrolling 

As explained earlier, only about half of the Success Academy lottery winners enrolled 
in Success Academy schools, and about a quarter of the students who lost their “local” Success 
Academy lotteries enrolled in different Success Academy schools. 18 

Table 7 presents the effects of enrolling in a Success Academy school on kindergarten 
entrants; on average they are between three to four times the effects of winning the opportunity 
to attend. For the kindergarten-entrant sample, the estimated enrollment effect was roughly 


^Approximately 50 percent of the lottery winners did enroll in a Success Academy school and roughly 25 
percent of those not selected crossed over into a Success Academy school (these rates are the same for 
kindergarten and first-grade applicants). The difference between these two estimates produces an estimated 
compliance rate for both kindergarten and first-grade applicants of between 25 and 30 percent. 
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Table 6 


The Effects of Winning a Success Academy Lottery 
on First-Grade Entrants After Three Years and After Four Years 


Outcome (%) 

Estimated 

Difference 

in Standard Deviations 

P-Value for 

Estimated 

Difference 

Third-grade test 

Mathematics 

0.185 * 

0.040 

Reading 

0.148 

0.097 

Fourth-grade test 

Mathematics 

0.267 ** 

0.003 

Reading 

0.078 

0.362 


SOURCES: MDRC's calculations use Success Academy school assignment data and 
New York City Department of Education school records and state test data for Success 
Academy applicants in the the spring of 2010. 

NOTES: This sample contains 314 lottery winners and 592 control group members. 

Values for the difference between lottery winners and control group members are 
obtained from a regression of a given outcome on a series of indicator variables that 
identify each lottery, plus an indicator variable that equals 1 for lottery winners and 0 
for lottery losers, plus a set of covariates: gender, age, free/reduced-price lunch status, 
race, and number of days absent in the baseline school year. The coefficient on the 
lottery winners indicator equals the difference in the mean follow-up outcome 
for lottery winners and control group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance 
levels are indicated as: ** = 1 percent; * = 5 percent. 


equivalent to an additional 1.1 years of math learning for this age group (effect size = 0.560, p- 
value = 0.086). 19 Said differently, at the end of the fourth follow-up year, Success Academy 
program group enrollees were a little over one year ahead of their control group counterparts in 
math. Kindergarten-entrant sample program group enrollees may also experience sizable 
improvements in reading (effect size = 0.3421, p-value = 0.185), but because there is considera¬ 
ble uncertainty around this estimated effect, as reflected in the nonsignificant p-value, one 
cannot rule out the possibility that the program did not have an effect in reading. 


19 Again, according to Hill, Bloom, Black, and Lipsey (2008), the typical annual gain for third-graders in 
math is 0.52 standard deviations; in reading it is 0.36 standard deviations. 
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Table 7 


The Effects of Enrolling in a Success Academy School 
on Kindergarten Entrants After Four Years 



Estimated 

P- Value for 


Difference 

Estimated 

Outcome (%) 

in Standard Deviations 

Difference 

Third-erade test 



Mathematics 

0.560 

0.086 

Reading 

0.341 

0.185 


SOURCES: MDRC's calculations use Success Academy school assignment data and 
New York City Department of Education school records and state test data for Success 
Academy applicants in the spring of 2010. 

NOTES: This sample contains 400 lottery winners and 3,001 control group members. 

Values for the difference between lottery winners and control group members are 
obtained from a 2sLs regression. 

A two-tailed t-test was applied to the estimated difference. Statistical significance 
levels are indicated as: ** = 1 percent; * = 5 percent. 


For the first-grade-entrant sample (Table 8), winning a lottery and enrolling in a Suc¬ 
cess Academy produces consistent third-grade and fourth-grade effects in math that are roughly 
equivalent in magnitude to an additional year to a year and a half of learning for this age group 
(effect size = 0.529, p-value = 0.040; and effect size = 0.763, p-value = 0.003, respectively). 20 
Said differently, at the end of the third follow-up year, Success Academy program group 
enrollees were roughly a year ahead of their control group counterparts in math; at the end of 
the fourth follow-up year they were 1.5 years ahead of their counterparts. These early math 
findings are comparable to some of the largest effects found in the charter field. 21 Success 
Academy’s effects on reading for the first-grade-entrant sample are promising but cannot be 
precisely estimated (effect size = 0.422, p-value = 0.097), especially in the fourth grade where, 
as discussed above, the result is far from statistically significant (effect size = 0.223, p-value = 
0.362). 


Success Academy produced these gains in the context of a control group that was al¬ 
ready performing well. Sixty percent of the control group members who did not attend Success 


‘"Again, according to Hill, Bloom, Black, and Lipsey (2008), the typical annual gain for fourth-graders in 
math is 0.56 standard deviations; in reading it is 0.40 standard deviations. 

21 Charbier, Cohodes, and Oreopoulos (2016). 
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Table 8 


The Effects of Enrolling in a Success Academy School 
on First-Grade Entrants After Three Years and After Four Years 


Outcome (%) 

Estimated 

Difference 

in Standard Deviations 

P-Value for 

Estimated 

Difference 

Third-erade test 

Mathematics 

0.529 * 

0.040 

Reading 

0.422 

0.097 

Fourth-grade test 

Mathematics 

0.763 ** 

0.003 

Reading 

0.223 

0.362 


SOURCES: MDRC's calculations use Success Academy school assignment data and 
New York City Department of Education school records and state test data for Success 
Academy applicants in the the spring of 2010. 

NOTES: This sample contains 314 lottery winners and 592 control group members. 

Values for the difference between lottery winners and control group members are 
obtained from a 2sLs regression. 

A two-tailed t-test was applied to the estimated difference. Statistical significance 
levels are indicated as: ** = 1 percent; * = 5 percent. 


Academy schools performed at or above grade level on their third-grade New York State math 
exams, and 44 percent performed at or above grade level on their third-grade New York State 
reading exams. These percentages are at least 10 points higher than the percentage of students 
scoring at or above grade level in the New York City district schools in the neighborhoods 
surrounding Success Academy schools. 


Conclusion 

Enrolling in a Success Academy school has positive effects on students’ math achievement and 
promising, though less clear, effects on students’ reading achievement. While the majority of 
the math effects are statistically significant at or near the level commonly used by education 
researchers (a p-value of 0.05 — or 5 percent), Success Academy’s effect on students’ reading 
achievement appears to be positive but cannot be precisely estimated. However, as previously 
discussed, the lottery sample available at the time of this analysis was relatively small, and had a 
low ratio of lottery “winners” to all applicants, which limits the sample’s statistical power; this 
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limited power may explain why the reading findings fall short of conventional standards of 
statistical significance. Additional research is necessary to better reveal Success Academy’s 
effects on students’ achievement in reading, and its effects on both reading and math in more 
recent years. 

Thus, there is a strong case for following students who applied in subsequent years and 
for adding a larger number of Success Academy schools to the sample. If the findings are 
sustained in a larger analysis, they would place Success Academy’s effects in the upper range of 
other charter schools. 22 Important questions MDRC has identified that have yet to be answered 
include: Once additional years of students are added to the sample and the statistical power of 
the analysis is increased, will the positive math findings be sustained and will there be a statisti¬ 
cally significant effect in reading? As Success Academy opens more schools across the district, 
will the characteristics of the students they serve change and will the effects at these newer 
schools be similar? 


22 Charbier, Cohodes, and Oreopoulos (2016). 
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Appendix A 


The Success Academy Admissions 
Process and the Models Applied in This Analysis 




Admission to every Success Academy school is managed centrally by the network’s director of 
enrollment. In the spring of 2010, Success Academy used four ranked “admissions priorities” 
based on whether applicants had siblings in Success Academy schools and based on where they 
lived. 1 From an applicant’s perspective, it may appear that there is one general admissions 
“lottery.” This appendix discusses the application process as it pertains to the lotteries MDRC 
has identified for research purposes, then describes the models used to estimate effects. 

When the school matching process begins in April, each student is assigned a random 
number that is used for all Success Academy schools to which she applies. Then, for each of 
these schools, she is placed in the admission-priority group that applies to her for that school 
and grade level. Within each admission-priority group for each Success Academy school, 
applicants are ordered by their random numbers. 

Each Success Academy school admits students, by grade level, in the order of their ad- 
missions-priority groups until all seats in a school are filled. Once a priority group is reached 
that is oversubscribed — that is, where the number of students in the priority group exceeds the 
number of remaining open seats in that grade at that school — the students in this priority group 
are admitted in the order of their random numbers until no more seats remain. 2 This process 
produces the statistical equivalent of a randomized lottery for students in that priority group for 
that school. As a result, the estimates of the effect of winning a Success Academy lottery are 
statistically equivalent to those of a randomized experiment. 

Since students can apply to multiple Success Academy schools, in any given year ap¬ 
plicants may be in multiple Success Academy lotteries. For example, families in the study 
sample applied to an average of six Success Academy schools (there were seven in operation 
that year). If an applicant was selected based on her random number in multiple Success 
Academies in school year 2010-2011, then she was offered a spot in the Success Academy 


'Each year these admissions priorities are predetermined and are the same for all Success Academy 
schools. According to its 2010 New York State charter application. Success Academy’s admission priorities 
were to admit applicants (in rank order): (1) who had siblings who already attended the Success Academy 
schools to which they were applying; (2) who were zoned to failing New York City Department of Education 
schools in the same school zones as the Success Academy schools to which they were applying; (3) who were 
zoned to failing New York City Department of Education schools outside of the school zones of the Success 
Academy schools to which they were applying; (4) whose parents’ addresses were within the zones of the 
Success Academy schools to which they were applying (also called those schools’ “Community School 
Districts”). 

2 The remaining students are placed on a waiting list, to be admitted in the order of their priority groups, 
and within each priority group in the order of their random numbers. Students are admitted from this waiting 
list as seats in the school become available. 
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school closest to her home address — that is, her “local” Success Academy school. 3 The present 
analysis is based on the results of the random selection for each student’s “local” Success 
Academy school. 4 This paper refers to students who competed in lotteries for their closest — 
“local” — Success Academy schools as “the sample of Success Academy lottery participants” 
and the lotteries they competed in as “Success Academy lotteries.” 


Estimating the Intent-to-Treat and Complier Average 
Causal Effects 

The intent-to-treat effect is estimated using the following ordinary least squares regression 
model: 


Y i = Iy=i aj * Lottery }i + /3 W * Win t + 8 x *X i + s i (1) 

where Y is student i’s third- or fourth-grade New York State standardized mathematics or 
reading test score (depending on the outcome for which effects are being estimated), Win is a 
lottery winner indicator equal to 1 if student i wins lottery j and 0 otherwise, Lottery is a vector 
of lottery indicators equal to 1 for lottery j and 0 otherwise, X is a vector of student-level 
covariates for student i (included for precision), and e is a random error for student i that is 
clustered by the first school that a student entered after a lottery. The parameter of interest, (3 W , 
identifies the effect of winning a lottery on student outcomes, and its standard error and associ¬ 
ated t-statistic identify its statistical significance. 

The first stage of this two-stage least squares estimation process is specified as: 

E t = Ey =1 ccj * Lotteryji + P W * Wiiii + 8 x *Xi+Wi (2) 

where E, is a Success Academy enrollment indicator equal to 1 if student i enrolled in a Success 
Academy school at any time before she took the state test used as an outcome for the analysis 


^Beginning in 2013, applicants had the opportunity to rank the Success Academy schools to which they 
applied, and if an applicant was selected based on her random number in multiple Success Academy schools, 
she was offered a spot in the school that she ranked highest among those schools for which she won a lottery. 

4 As discussed in the section below, in the main analysis, if a student lost a lottery for the Success Acade¬ 
my school closest to where she lived, the student is considered a control group member, even if she won a 
lottery for a different Success Academy school or came off the waiting list for the nearest school at a later time. 
In the analysis of the effects of enrolling in a Success Academy school, if a student won a lottery for a different 
Success Academy school and enrolled there, that student would be considered a “control group crossover.” 

Another commonly applied approach to analyzing extant lottery data — grouping students based on the 
set of schools they applied to — is not possible in this context because there are too many possible combina¬ 
tions of the seven schools selected by applicants. 
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and 0 otherwise. All other terms are defined as in equation (1). The second-stage equation is 
specified as: 

Y i = S-E i + £y =1 (Xj ■ Lottery n + 6-X i + e i (3) 

where Ej equals the fitted value of the enrollment outcome from the first-stage equation, and 
ej is a random error that is clustered by the schools that students entered after their lotteries. The 
estimated value of 5 is a consistent estimate of the average effect of enrolling in Success 
Academy schools for target Success Academy enrollees. 
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